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DETAILED ACTION 

This action is a Final in response to annendnnent/rennarl<s filed on 03/21/2008. 

Claims 1,4-5, 8-13, 16-17, 20-25, 28-29, 32-36, and 44-52, are pending. Claims 
44-45 are currently amended to correct the typographical errors and dependency errors, 
which set forth in the previous office action dated 12/21/2007. Claims 2-3, 6-7, 14-15, 
18-19, 26-27, 30-31, and 37-43 are previously canceled; thus the objection to claim 45 
is hereby withdrawn. Claims 1,4-5, 8-9, 1,13, 16-17, 20-21, 23, 25, 28-29, 32-33, 35, 
and 46-52 were previously presented. Claims 10, 12, 22, 24, 34, and 36 were original 
presented. Effective filing date is 03/15/2000, (Assignee: Ricoh). 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as 
set forth in section 102 of this title, if the differences between the subject matter sought to be 
patented and the prior art are such that the subject matter as a whole would have been obvious 
at the time the invention was made to a person having ordinary skill in the art to which said 
subject matter pertains. Patentability shall not be negatived by the manner in which the invention 
was made. 

Claims 1, 4-5, 8-11, 13, 16-17, 20-23, 25, 28-29, 32-35, and 44-52 are rejected 
under 35 U.S.C. 103(a) as being as being unpatentable over Morris et al 
US005420974A issued 05/30/1 995 (hereinafter Morris), in view of Schena et al 
US006448979B1 filed 01/25/1999 (hereinafter Schena). 
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Regarding independent claim 1: Morris teaclies: 

a method comprising: creating a multimedia annotation for a paper 
document, the multimedia annotation representing at least one of an audio 
and a video clip, 

(See Morris at Fig/ 9A-B and Col. 2, Lines 35-40, discloses scanned-in form, and are 
stored in a mixed object document content architecture envelope (MODCA). Morris 
further discloses, if the box being defined is a check box, then the sub-type is defined as 
audio, video, image, text, or other type object. The resulting directory and the image 
object of the overall scanned-in form are stored in a mixed object document content 
architecture envelope (MODCA)-see Morris at Col. 2 Lines 20-30.) 
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creating a first multimedia document by combining the paper 
document and the multimedia annotation represented by at least one of the 
audio sound and video clip, 

(See Morris at Col. 2, Lines 35-40, discloses scanned-in form, are stored in a mixed 
object document content architecture envelope (MODCA). Morris further discloses, if 
the box being defined is a check box, then the sub-type is defined as audio, video, 
image, text, or other type object. The resulting directory and the image object of the 
overall scanned-in form are stored in a mixed object document content architecture 
envelope (MODCA)-see Morris at Col. 2 Lines 20-30.) 

wherein the first multimedia document is generated as a part of 
reproducing the paper document via a document reproduction system, 
wherein the multimedia annotation is captured via an input device of the 
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document reproduction system while the paper document is being 
reproduced via the document reproduction system, wherein the captured 
multimedia annotation is encoded, 

(See Morris at Col. 2, Lines 35-40, discloses scanned-in form, are stored in a mixed 
object document content architecture envelope (MODCA). Morris further discloses, if 
the box being defined is a check box, then the sub-type is defined as audio, video, 
image, text, or other type object. The resulting directory and the image object of the 
overall scanned-in form are stored in a mixed object document content architecture 
envelope (MODCA)-see Morris at Col. 2 Lines 20-30. 

Also see Morris at Col. 2, Lines 25-40, discloses a filled-out hard copy of the form 
is desired to be entered into the image archiving system, the filled-out copy of the 
master form is scanned-in to the system. A special indicium is associated with each 
master form, which uniquely identifies that form and distinguishes it from all other 
master forms created for the system. The system reads the indicium from the scanned- 
in form and then accesses the MODCA envelope the directory for the master form is 
then accessed to obtain the coordinates of each component field on the scanned-in 
form. Then access the stored nonvisual object and play it back for the operator- See 
Morris at the Abstract.) 

wherein the first multimedia document, which when scanned by a process, 
the process causes the printed multimedia annotation to be decoded, the at 
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least one of the audio sound and video clip to be extracted from the 
multimedia annotation, 

(See Morris at Col. 2, Lines 25-40, discloses a filled-out hard copy of the form is desired 
to be entered into the image archiving system, the filled-out copy of the master form is 
scanned-in to the system. A special indicium is associated with each master form, which 
uniquely identifies that form and distinguishes it from all other master forms created for 
the system. The system reads the indicium from the scanned-in form and then accesses 
the MODCA envelope the directory for the master form is then accessed to obtain the 
coordinates of each component field on the scanned-in form.) 

and the at least one extracted audio sound and video clip can be played via 
a multimedia player. 

(See Morris at Col. 3, Lines 35-45, discloses when the check box is selected, the 
multimedia record is accessed from the MODCA envelope, for example, and is played. 
If it is an audio record, then the sound is played for the operator. If it is a video record, 
then a video cameo display window can be presented to the operator on the display 
screen and the video sequence played (i.e. played via a multimedia player). 

In addition. Morris does not teach, but Schena teaches: 

multimedia annotation represented by a first bar code, 
wherein the captured multimedia annotation is encoded within the first bar 
code. 
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(See Schena at Col. 3 Lines 60-65, discloses high density barcodes. 

Also see Schena Col. 1 lines 10-25, discloses multinnedia information (i.e. audio, 
video) using a scanner for machine-readable code containing a link information 
corresponding to a provider information depicted on the printed medium). 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify Morris's check box and the sub-type is defined as audio, video, 
image, text, or other type object scanned-in form, are stored in a mixed object document 
content architecture envelope (MODCA), to include a means of utilizing the barcode as 
taught by Schena, so that, when selected by a user (i.e. scanned document with the box 
is check as taught by Morris ) the barcode would accessing a segment of video file 
which references scanned document, and provides a predictable result of the bridging 
the gap between the virtual multimedia-based of the Internet word and the physical 
world of printed media, wherein the annotate media may contain one or more of textual, 
audio, or video information, and the customer premises equipment ("CPE"), serving as 
the receiver, plays multimedia sequence (Schena col.1, lines 5-15, col. 1 line 64 
through col. 2. line 40). 

Regarding independent claim 13, 

Is directed to machine-readable medium providing instructions, which 
when executed by a set of one or more processors to perform the method of 
claim 1 which cites above, and is similarly rejected under the same rationale. 
Also See Morris at Fig. 1 and also at Col. 5, Lines 10-20. 
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Regarding independent claim 25, 

Is directed to a computer system include a bus, a data storage, and a 
processor to perform the method of claim 1 which cites above, and is similarly 
rejected under the same rationale. Also See Morris at Fig. 1 and also at Col. 5, 
Lines 10-20. 

Regarding claims 4, 16, and 28, Morris teaches: 

wherein a location indicator associated with the multimedia 
annotation is placed on the first multimedia document, wherein the location 
indicator indicates where the multimedia annotation can be retrieved and 
played. 

(See Morris at Col. 3, Lines 35-45, discloses when the check box is selected, the 
multimedia record is accessed from the MODCA envelope, for example, and is played. 
If it is an audio record, then the sound is played for the operator. If it is a video record, 
then a video cameo display window can be presented to the operator on the display 
screen and the video sequence played (i.e. played via a multimedia player). 

Regarding claims 5, 17, and 29, 

Morris does not expressly teach, but Schena teaches: 

wherein the location indicator comprises a first Uniform Resource 
Locator (URL), and a second bar code, wherein the first URL is indicated in 
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plain text, and wherein the second bar code represents the first URL in an 
encrypted form. 

(Schena col. 4, lines 45-55, discloses the link information corresponding to the provider 
information may include a universal resource locator (URL) (Schena col.1, line 64 
through col. 2. line 40). In addition, the links can be encoded according to provider, for 
example UPC or ISBN numbers and any code may serve as the alphanumeric 
sequence. 

Also See Schena at Col. 3 Lines 60-65, discloses high density barcodes. 
Under the broadest reasonable interpretation. The Examiner equates the claimed bar 
code as equivalent to the UPC or ISBN numbers and any code as taught by Schena. 
This interpretation is supported by Applicant's Specification, which states "The 
annotation may be in different forms, sucli as, for example, a bar code containing an 
audio message or a URL indicating a linl< to a video clip" at page 7 lines 15-18. 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify schena's a user selectable object (i.e. the location indicator 
comprises a first Uniform Resource Locator (URL), and a second bar code, wherein the 
first URL is indicated in plain text, and wherein the second bar code represents the first 
URL in an encrypted form) providing a user with access to the portion of the recorded 
information corresponding to the visual feature (i.e. machine readable barcode), to 
includes a means of selecting by a user onto a check box, the sub-type is defined as 
audio, video, image, text, or other type object scanned-in form, are stored in a mixed 
object document content architecture envelope (MODCA) as taught by Morris. One of 
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ordinary sl<ill in tlie art would liave been motivated to modify tliis combination because 
Scliena and Morris are from tlie same field of endeavor of electronic presentation 
includes annotation to presentation material, and provides a predictable result of the 
bridging the gap between the virtual multimedia-based of the Internet word and the 
physical world of printed media, wherein the annotate media may contain one or more 
of textual, audio, or video information, and the customer premises equipment ("CPE"), 
serving as the receiver, plays multimedia sequence (Schena col.1, lines 5-15, col. 1 
line 64 through col. 2. line 40). 

Regarding claims 8, 20, and 32, Morris teaches: 

creating a second multimedia document by combining the image of 
the paper document and the multimedia annotation; and storing the image 
of the paper document and the multimedia annotation in a storage, wherein 
the second multimedia document is an electronic document associated 
with the first multimedia document which is a physical document. 
(See Morris at Col. 2, Lines 35-40, discloses scanned-in form, are stored in a mixed 
object document content architecture envelope (MODCA). Morris further discloses, if 
the box being defined is a check box, then the sub-type is defined as audio, video, 
image, text, or other type object. The resulting directory and the image object of the 
overall scanned-in form are stored in a mixed object document content architecture 
envelope (MODCA)-see Morris at Col. 2 Lines 20-30. 
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Also see Morris at Col. 2, Lines 25-40, discloses a filled-out hard copy of the form 
is desired to be entered into the image archiving system, the filled-out copy of the 
master form is scanned-in to the system. A special indicium is associated with each 
master form, which uniquely identifies that form and distinguishes it from all other 
master forms created for the system. The system reads the indicium from the scanned- 
in form and then accesses the MODCA envelope the directory for the master form is 
then accessed to obtain the coordinates of each component field on the scanned-in 
form. Then access the stored nonvisual object and play it back for the operator- See 
Morris at the Abstract.) 

generating an image of the paper document, the image of the paper 
document being unconsciously captured via the document reproduction 
system during the reproduction of the paper document without user 
intervention; 

(See Morris at Col. 2, Lines 25-40, discloses a filled-out hard copy of the form is desired 
to be entered into the image archiving system, the filled-out copy of the master form is 
scanned-in to the system. A special indicium is associated with each master form, which 
uniquely identifies that form and distinguishes it from all other master forms created for 
the system. The system reads the indicium from the scanned-in form and then accesses 
the MODCA envelope the directory for the master form is then accessed to obtain the 
coordinates of each component field on the scanned-in form. Then access the stored 
nonvisual object and play it back for the operator- See Morris at the Abstract.) 
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Regarding claims 9, 21, and 33, 

Morris does not expressly teach, but Schena teaches: 

wherein a multimedia document is represented as a second Uniform 

Resource Locator (URL) printed on the first multimedia document, and 

wherein the image of the paper document and the multimedia annotation is 

accessed with the second URL, 
(Schena col. 4, lines 45-55, discloses the link information corresponding to the provider 
information may include a universal resource locator (URL) (Schena col.1, line 64 
through col. 2. line 40). In addition, the links can be encoded according to provider, for 
example UPC or ISBN numbers and any code may serve as the alphanumeric 
sequence. 

Also See Schena at Col. 3 Lines 60-65, discloses high density barcodes. 
Under the broadest reasonable interpretation. The Examiner equates the claimed bar 
code as equivalent to the UPC or ISBN numbers and any code as taught by Schena. 
This interpretation is supported by Applicant's Specification, which states "The 
annotation may be in different forms, sucli as, for example, a bar code containing an 
audio message or a URL indicating a linl< to a video clip" at page 7 lines 15-18.) 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify schena's a user selectable object (i.e. the location indicator 
comprises a first Uniform Resource Locator (URL), and a second bar code, wherein the 
first URL is indicated in plain text, and wherein the second bar code represents the first 
URL in an encrypted form) providing a user with access to the portion of the recorded 



Application/Control Number: 09/526,031 Page 13 

Art Unit: 2176 

information corresponding to tlie visual feature (i.e. macliine readable barcode), so that, 
when selected by a user the barcode would accessing a segment of video file which 
references scanned document as taught by Morris. One of ordinary skill in the art would 
have been motivated to modify this combination because Schena and Morris are from 
the same field of endeavor of electronic presentation includes annotation to 
presentation material, and provides a predictable result of the bridging the gap between 
the virtual multimedia-based of the Internet word and the physical world of printed 
media, wherein the annotate media may contain one or more of textual, audio, or video 
information, and the customer premises equipment ("CPE"), serving as the receiver, 
plays multimedia sequence (Schena col.1, lines 5-15, col. 1 line 64 

Regarding claims 10, 22, and 34, 

Morris does not expressly teach, but Schena teaches: 
wherein a third bar code is used to represent a second URL 

(Schena col. 4, lines 45-55, discloses the link information corresponding to the provider 
information may include a universal resource locator (URL) (Schena col.1, line 64 
through col. 2. line 40). In addition, the links can be encoded according to provider, for 
example UPC or ISBN numbers and any code may serve as the alphanumeric 
sequence. 

Also See Schena at Col. 3 Lines 60-65, discloses high density barcodes. 
Under the broadest reasonable interpretation. The Examiner equates the claimed bar 
code as equivalent to the UPC or ISBN numbers and any code as taught by Schena. 
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This interpretation is supported by Applicant's Specification, wliicli states "The 
annotation may be in different forms, sucli as, for example, a bar code containing an 
audio message or a URL indicating a linl< to a video clip" at page 7 lines 15-18.) 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify schena's a user selectable object (i.e. the location indicator 
comprises a first Uniform Resource Locator (URL), and a second bar code, wherein the 
first URL is indicated in plain text, and wherein the second bar code represents the first 
URL in an encrypted form) providing a user with access to the portion of the recorded 
information corresponding to the visual feature (i.e. machine readable barcode), so that, 
when selected by a user the barcode would accessing a segment of video file which 
references scanned document as taught by Morris. One of ordinary skill in the art would 
have been motivated to modify this combination because Schena and Morris are from 
the same field of endeavor of electronic presentation includes annotation to 
presentation material, and provides a predictable result of the bridging the gap between 
the virtual multimedia-based of the Internet word and the physical world of printed 
media, wherein the annotate media may contain one or more of textual, audio, or video 
information, and the customer premises equipment ("CPE"), serving as the receiver, 
plays multimedia sequence (Schena col.1, lines 5-15, col. 1 line 64 
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Regarding claims 11, 23, and 35, Morris teaclies: 

wherein the recipient is specified by a user via an interface of the 
document reproduction system when the user reproduces the paper 
document using the document reproduction system. 

(See Morris at Col. 2, Lines 35-40, discloses scanned-in form, are stored in a mixed 
object document content architecture envelope (MODCA). Morris further discloses, if 
the box being defined is a check box, then the sub-type is defined as audio, video, 
image, text, or other type object. The resulting directory and the image object of the 
overall scanned-in form are stored in a mixed object document content architecture 
envelope (MODCA)-see Morris at Col. 2 Lines 20-30.) 

In addition. Morris does not expressly teach, but Schena teaches: 

automatically sending a second multimedia document to a recipient 
by electronic mail as a part of reproducing the paper document via the 
document reproduction system, 

(See Schena at Col. 4 lines 35, discloses the link information was published or located, 
along with message-specific information. The examiner equates the claimed 
automatically sending a second multimedia document to a recipient by electronic mail 
as equivalent to link information was published or located, along with a message as 
taught by Schena.) 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify schena's automatically sending a second multimedia document to a 
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recipient by electronic mail as a part of reproducing the paper document, so that, a user 
can accessing a segment of video file which references scanned document as taught by 
Morris. One of ordinary skill in the art would have been motivated to modify this 
combination because Schena and Morris are from the same field of endeavor of 
electronic presentation includes annotation to presentation material, and provides a 
predictable result of the bridging the gap between the virtual multimedia-based of the 
Internet word and the physical world of printed media, wherein the annotate media may 
contain one or more of textual, audio, or video information, and the customer premises 
equipment ("CPE"), serving as the receiver, plays multimedia sequence (Schena col.1, 
lines 5-15, col. 1 line 64 through col. 2. line 40). 

Regarding claim 44, Morris teaches: 

wherein the first multimedia document is a physical document 
having the first check box printed thereon, which when scanned by a 
scanning device, causes the first check box to be decoded and the audio 
sound to be extracted from the first check box, and wherein the extracted 
audio sound is capable of being played via an audio player. 
(See Morris at Col. 3, Lines 35-45, discloses when the check box is selected, the 
multimedia record is accessed from the MODCA envelope, for example, and is played. 
If it is an audio record, then the sound is played for the operator. If it is a video record, 
then a video cameo display window can be presented to the operator on the display 
screen and the video sequence played (i.e. played via a multimedia player). 
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In addition. Morris does not expressly teach, but Schena teaches: 
bar code, 

(See Schena at Col. 3 Lines 60-65, discloses high density barcodes.) 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify Morris's check box and the sub-type is defined as audio, video, 
image, text, or other type object scanned-in form, are stored in a mixed object document 
content architecture envelope (MODCA), to include a means of utilizing the barcode as 
taught by Schena, so that, when selected by a user (i.e. scanned document with the box 
is check as taught by Morris ) the barcode would accessing a segment of video file 
which references scanned document, and provides a predictable result of the bridging 
the gap between the virtual multimedia-based of the Internet word and the physical 
world of printed media, wherein the annotate media may contain one or more of textual, 
audio, or video information, and the customer premises equipment ("CPE"), serving as 
the receiver, plays multimedia sequence (Schena col.1, lines 5-15, col. 1 line 64 
through col. 2. line 40). 

Regarding claim 45, Morris teaches: 

capturing an audio sound of the multimedia annotation from a user 
using a microphone of the input device to annotate the paper document to 
create a multimedia paper document. 

(See Morris at Col. 2, Lines 5-20, discloses sound records. Video records, sequential 
animation sequence records, and the like, such multimedia objects being characterized 
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in tliat tlieir presentation requires duration of time. It is witliin tlie scope of tlie invention 
tliat cliecl< boxes can also be used to represent visual fields such as other image 
objects, other text objects, other vector graphics objects, etc. Morris at Col. 3 Lines 10- 
15, discloses the steps that allow a user to add new recorded segments of audio, video, 
etc. for any page of data. It is inherently including the use of Microphone and Video 
camera in order to record sound and Video as taught by Morris.) 

Regarding claim 46, Morris teaches: 

wherein the microphone is automatically activated when the user 
selects a reproduction function of the document reproduction system to 
reproduce the paper document. 

(See Morris at Col. 2, Lines 17 -60, discloses sound records. Video records, sequential 
animation sequence records, and the like, such multimedia objects being characterized 
in that their presentation requires duration of time. It is within the scope of the invention 
that check boxes can also be used to represent visual fields such as other image 
objects, other text objects, other vector graphics objects, etc. Morris at Col. 3 Lines 10- 
15, discloses the steps that allow a user to add new recorded segments of audio, video, 
etc. for any page of data. It is inherently including the use of Microphone and Video 
camera in order to record sound and Video as taught by Morris. Also Morris discloses if 
the object is a nonvisual field object, such as a voice object, then the operator is 
prompted to input the voice record. Also, the audio check box corresponding to the 
audio record is highlighted- see Morris at Col. 4 Lines 1-15.) 
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Claim 47, Morris teaclies: 

capturing a video clip of the multimedia annotation from a user 
using a video camera of the input device to annotate the paper document 
to create a multimedia paper document. 

(See Morris at Col. 2, Lines 5-20, discloses sound records. Video records, sequential 
animation sequence records, and the like, such multimedia objects being characterized 
in that their presentation requires a duration of time. It is within the scope of the 
invention that check boxes can also be used to represent visual fields such as other 
image objects, other text objects, other vector graphics objects, etc. Morris at Col. 3 
Lines 10-15, discloses the steps that allow a user to add new recorded segments of 
audio, video, etc. for any page of data. It is inherently including the use of Microphone 
and Video camera in order to record sound and Video as taught by Morris.) 

Claim 48, Morris teaches: 

wherein the video camera is automatically activated when the user 
selects a reproduction function of the document reproduction system to 
reproduce the paper document. 

(See Morris at fig. 9 and Col. 9, Line 45 through Col, 10, Line 5, discloses Step 1040 
determines whether a video check box has been marked. In the example shown in FIG. 
9A, the video check box 04 has been marked with the mark X which is reference 60. 
Step 1040 then prompts the operator to input the video record. Step 1042 forms a video 
object content architecture (VDOCA) object 58. Step 1044 stores the object 58 in the 
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MODCA envelope 40, as is sliown in FIG. 9B and adds the object name " VIDEO-1" as 
the pointer in the check box 04 line of the directory 42C (Table 5). It then adds the 
object name " VIDEO-1" and the media address " MODCA-VOCA" in the video data line 
of the directory 42C (Table 5). It is inherently including the use of Microphone and Video 
camera in order to record sound and Video as taught by Morris.) 

Claim 49, Morris teaches: 

in response to a request to retrieve a second multimedia document, 
performing a content-based search for the requested multimedia document 
within the storage based on the content of the multimedia annotation 
associated with the requested multimedia document. 
(See Morris at Col. 3, Lines 35-45, discloses when the check box is selected, the 
multimedia record is accessed from the MODCA envelope, for example, and is played. 
If it is an audio record, then the sound is played for the operator. If it is a video record, 
then a video cameo display window can be presented to the operator on the display 
screen and the video sequence played (i.e. played via a multimedia player). 

Claims 50-52, Morris teaches: 

wherein the content-based search is performed by OCR, audio 
speech recognition, or video face recognition techniques on the 
multimedia annotations of the multimedia documents being searched. 
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(See Morris at Col. 2, Lines 35-40, discloses scanned-in form (I.e. OCR), are stored in a 
mixed object document content architecture envelope (MODCA). Morris further 
discloses, if the box being defined is a check box, then the sub-type is defined as audio, 
video, image, text, or other type object. The resulting directory and the image object of 
the overall scanned-in form are stored in a mixed object document content architecture 
envelope (MODCA)-see Morris at Col. 2 Lines 20-30.) 

Claims 12, 24, and 36 are rejected under 35 U.S.C. 103(a) as being as being 
unpatentable over Morris et al US005420974A issued 05/30/1995 (hereinafter Morris), 
in view of Schena et al US006448979B1 filed 01/25/1999 (hereinafter Schena), and 
further in view of Hallida v et al., US 5,880,740 filed 7/12/1996 (hereinafter Halliday). 

Regarding claims 12, 24, and 36, 

Morris, and Schena do not teach, but Hallidav teaches: 

wherein the recipient receives the image of the paper document and 
the multimedia annotation in the form of Multi-purpose Internet Mail 
Extension (MIME). 

(See Halliday at Col. 8 Lines 5-28, discloses MIME.) 

It would have been obvious to one of ordinary skill in the art at the time of the 
invention to modify Morris and Schena teaching of using a scanner capable of receiving 
data such as machine-readable code from a printed medium, so that, when selected by 
a user the barcode would accessing a segment of video file which references scanned 
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document as taught by Morris, to includes MIME functionality as taught by Halliday. One 
of ordinary skill in the art would have been motivated to modify this combination provide 
a predictable result of the bridging the gap between the virtual multimedia-based of the 
Internet word and the physical world of printed media, wherein the annotate media may 
contain one or more of textual, audio, or video information, and the customer premises 
equipment ("CPE"), serving as the receiver, plays multimedia sequence (Schena col.1, 
lines 5-15, col. 1 line 64 through col. 2. line 40), and provides a composite document as 
a MIME attachment to an email message and sent via an conventional email utility, 
such as the popular Eudora, America Online, or Netscape Navigator mail programs (see 
Halliday at Col. 7, Line 39, and Col. 8 Lines 10-15). 

It is noted that any citations to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the references should not be considered to 
be limiting in any way. A reference is relevant for all it contains and may be relied upon 
for all that it would have reasonably suggested to one having ordinary skill in the art. 
See, MPEP2123. 
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Response to Arguments 

Brief description of cited prior art: 

Morris [[Hereinafter Morris] discloses scanned-in form, are stored in a mixed object 
document content arcliitecture envelope (MODCA), at Col. 3, lines 35-45. Morris further 
discloses at the Abstract, a multimedia document form system enables the definition, 
manipulation, storage and editing of a form which contains multimedia objects. For 
example an audio object will be entered in digitized form and stored in association with 
the form. When the form is then displayed on a visual display device, the corresponding 
check box is highlighted to indicate to the operator the presence of a nonvisual 
presentation object which is stored in association with the form. The operator can then 
select the check box with a mouse, and the system will then access the stored 
nonvisual object and play it back for the operator. See the Abstract. 

Schena [hereinafter Schena] discloses the machine-readable code on the paper is used 
to communicate corresponding multimedia information when the machine-readable 
code is read by a scanner. The URL encoded in the machine-readable code points to a 
multimedia file. Thus, the machine-readable code on the printed medium annotates the 
printed medium with the referenced multimedia file, thus combining the printed medium 
with virtual multimedia. In combining the multimedia presentation with the printed 
medium, Schena has created a multimedia document of the claimed invention (See 
Schena the Abstract) and also Schena in col. 2 lines 6-8 that the multimedia annotation 
may contain one or more of textual, audio, or video information. 
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Beginning on page 12 of tlie Rennarl<s (liereinafter tlie rennarl<s), Applicant 
argues tlie following issues, which are accordingly addressed below. 

Rejection of Claims 1, 4-5, 8-11, 13, 16-17, 20-23, 25, 28-29, 32-35, and 44-52 
Under 35 U.S.C. § 103(a) over Morris in view of Schena: 

Firstly: Applicant argues, the combination of Morris and Schena fail to teach "a 
multimedia document witli a barcode encoding audio and/or video clips sucli tliat tlie 
audio and/or video clips may be subsequently extracted and played from tlie barcode, " 
because Morris and Schena do not teach or suggest: "encode" at least the one of an 
audio sound and a video clip within a first barcode, but rather describes that audio, 
video, or animation associated with a document is stored in a storage directory for the 
document, and a pointer in the document points to the storage location. See the 
remarks Page 14, the second and third para). 

The examiner disagrees. 

For purposes of responding to Applicant's argument, the examiner will assume 
that Applicant is arguing for the patentability of Claim 1 . 

As discuss in the rejection above, specifically Morris discloses scanned-in form, 
are stored in a mixed object document content architecture envelope (MODCA), at Col. 
3, lines 35-45. Morris further discloses at the Abstract, a multimedia document form 
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system enables the definition, manipulation, storage and editing of a form which 
contains multimedia objects. For example an audio object will be entered in digitized 
form and stored in association with the form. When the form is then displayed on a 
visual display device, the corresponding check box is highlighted to indicate to the 
operator the presence of a nonvisual presentation object which is stored in association 
with the form. The operator can then select the check box with a mouse, and the system 
will then access the stored nonvisual object and play it back for the operator. See the 
Abstract. 

In addition, Schena discloses, the machine-readable code on the paper is used 
to communicate corresponding multimedia information when the machine-readable 
code is read by a scanner .The URL encoded in the machine-readable code points to a 
multimedia file. Thus, the machine-readable code on the printed medium annotates the 
printed medium with the referenced multimedia file, thus combining the printed medium 
with virtual multimedia. In combining the multimedia presentation with the printed 
medium, Schena has created a multimedia document of the claimed invention (See 
Schena the Abstract) and also Schena in col. 2 lines 6-8 that the multimedia annotation 
may contain one or more of textual, audio, or video information. Also, Schena further 
discloses multimedia annotation represented by a first bar code, wherein the captured 
multimedia annotation is encoded within the first bar code, in that the multimedia 
information (i.e. audio, video) using a scanner for machine-readable code containing a 
link information corresponding to a provider information depicted on the printed medium. 
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see Schena Col. 1 lines 10-25. Also Schena at Col. 3 Lines 60-65, discloses high 
density barcodes. 

Accordingly, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify Morris's check box and the sub-type is defined as audio, 
video, image, text, or other type object scanned-in form, are stored in a mixed object 
document content architecture envelope (MODCA), to include a means of utilizing the 
barcode as taught by Schena, so that, when selected by a user (i.e. scanned document 
with the box is check as taught by Morris ) the barcode would accessing a segment of 
video file which references scanned document, and provides a predictable result of the 
bridging the gap between the virtual multimedia-based of the Internet word and the 
physical world of printed media, wherein the annotate media may contain one or more 
of textual, audio, or video information, and the customer premises equipment ("CPE"), 
serving as the receiver, plays multimedia sequence (Schena col.1, lines 5-15, col. 1 
line 64 through col. 2. line 40). 

Therefore, Morris and Schena clearly teach a multimedia document with a 
barcode encoding audio and/or video clips such that the audio and/or video clips may 
be subsequently extracted and played from the barcode as claimed. 

Secondly: Applicant argues, "The link described by Schena, whether it is a standard, 
enhanced, or high density barcode, only provides a pointer to a location where the 
encoded annotation may be obtained. The link, itself, however is not described by 
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Schena as being encoded sucli tliat it can be played as a voice or video clip in a media 
player." See the remarks Page 14 fourth para ^ Page 15). 

The examiner disagrees. 

For purposes of responding to Applicant's argument, the examiner will assume 
that Applicant is arguing for the patentability of Claim 1 . 

As discuss in the rejection above. Firstly, the Applicant invention specification 
discloses, "The annotation may be in different forms, sucli as, for example, a bar code 
containing an audio message or a URL indicating a link to a video clip." (Applicants 
invention specification page 7 lines 15-18). Where the Examiner interpreted the claimed 
in light of the specification, thus it would have been obvious to one of ordinary skill in the 
art to substitute applicant's barcode with applicant's URL indicating a link to a video clip 
to provide access to a video clip. Secondly, Applicant admitted, "f/?e multimedia 
annotation may be a barcode", the phase "may be" rendering the uncertainty of choice, 
which is not limited only to the barcode and not to a URL indicating a link to a video clip. 

To clarify, the Examiner introduce Morris reference as discuss in the rejection 
above, specifically, discloses an audio object will be entered in digitized form and stored 
in association with the form. When the form is then displayed on a visual display device, 
the corresponding check box is highlighted to indicate to the operator the presence of a 
nonvisual presentation object which is stored in association with the form. The operator 
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can then select the check box with a mouse, and the system will then access the stored 
nonvisual object and play it back for the operator. See the Abstract. 

Accordingly, it would have been obvious to one of ordinary skill in the art at the 
time of the invention to modify Morris's check box and the sub-type is defined as audio, 
video, image, text, or other type object scanned-in form, are stored in a mixed object 
document content architecture envelope (MODCA), to include a means of utilizing the 
barcode as taught by Schena, so that, when selected by a user (i.e. scanned document 
with the box is check as taught by Morris ) the barcode would accessing a segment of 
video file which references scanned document, and provides a predictable result of the 
bridging the gap between the virtual multimedia-based of the Internet word and the 
physical world of printed media, wherein the annotate media may contain one or more 
of textual, audio, or video information, and the customer premises equipment ("CPE"), 
serving as the receiver, plays multimedia sequence (Schena col.1, lines 5-15, col. 1 
line 64 through col. 2. line 40). 

Therefore, Morris and Schena clearly teach a multimedia document with a 
barcode encoding audio and/or video clips such that the audio and/or video clips may 
be subsequently extracted and played from the barcode as claimed. 

Thirdly: Applicant argues that Morris fail to teach, "multimedia annotation is captured 
via an input device of tlie document reproduction system wliile tlie paper document is 
being reproduced via tlie document reproduction system" because Morris merely 
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receives an address wliere content is stored, 
device. See tlie rennarl<s Page 16-17. 



Page 29 

but does not capture tlie content via input 



Tlie examiner disagrees. 

For purposes of responding to Applicant's argument, tlie examiner will assume 
that Applicant is arguing for the patentability of Claim 1 . 

As discuss in the rejection above, as discuss in the rejection above, specifically 
Morris discloses scanned-in form (capture via input device), are stored in a mixed object 
document content architecture envelope (MODCA), at Col. 3, lines 35-45. Morris further 
discloses at the Abstract, a multimedia document form system enables the definition, 
manipulation, storage and editing of a form which contains multimedia objects. For 
example an audio object will be entered in digitized form and stored in association with 
the form. When the form is then displayed on a visual display device, the corresponding 
check box is highlighted to indicate to the operator the presence of a nonvisual 
presentation object which is stored in association with the form. The operator can then 
select the check box with a mouse, and the system will then access the stored 
nonvisual object and play it back for the operator (reproducing via reproduction system). 
See the Abstract. 

Thus, Morris and Schena clearly teach multimedia annotation is captured via an 
input device of the document reproduction system while the paper document is being 
reproduced via the document reproduction system as claimed. 
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Accordingly, for at least all the above evidence claims 1, 4-5, 8-11, 13, 16-17, 
20-23, 25, 28-29, 32-35, and 44-52 remain rejected at least at this time. 



Conclusion 

Accordingly THIS ACTION IS MADE FINAL See MPEP § 706.07(a). Applicant is 
reminded of the extension of time policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Quoc A. Tran whose telephone number is 571-272- 
8664. The examiner can normally be reached on Mon through Fri SAM - 5PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Doug Hutton can be reached on (571)272-4137. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding tlie status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

/Quoc A, Tran/ 
Patent Examiner 



/Rachna Desai/ 
Primary Examiner, AU 2176 



